Digital images are becoming an increasingly popular form of infoimaging. One reason for this popularity is the ease with which users can manipulate, edit, alter and enhance such digital images. For example, users often use manual digital imaging and editing tools, such as the crop and zoom tools provided in the Kodak Picture CD software sold by Eastman Kodak Company, Rochester, N.Y., U.S.A., to improve the appearance of digital images. These image editing tools allow a user to limit the image content of an image to emphasize important elements in the image. Other image editing tools can also be usefully applied to portions of images that a user considers to be important. However, manual editing tools require that the user manually designate areas of importance in each image that is to be edited. Many users find this process time consuming and, accordingly, only select images are edited in this manner.
Automatic and semi-automatic image processing and editing algorithms are known. These can be applied to enhance the appearance of a digital image while requiring little user input. These automatic and semi-automatic image processing algorithms analyze the content of an image and apply various assumptions about what the user would likely find to be important elements of an image. For example, large oval shaped objects having color that approximates known flesh tones can be assumed to be important to the user. The degree of presumed importance can be increased where, for example, the large oval face shaped objects are positioned near the center of an image. See for example, commonly assigned U.S. Pat. No. 6,282,317, entitled “Method For Automatic Determination of Main Subjects in Photographic Images” filed by Luo et al. on Dec. 31, 1998. Other algorithms use frequency analysis of the digital data that forms digital images to identify elements of an image that are considered to be of greater importance. Such algorithms make assumptions about what is important in an image based upon analysis of the visual elements of the captured image. See for example commonly assigned U.S. patent application Ser. No. 09/176,805 entitled “Determining Portions of a Digital Image Which are In Focus” filed by Erkkilea et al. on Oct. 22, 1998.
Knowledge of what is important in an image can be useful for other purposes. For example, when searching for images, a user manually sorts through images or manually inputs text based descriptions of images to enable an image search. What is preferred of course, is for the user to submit an exemplar image from which similar images can be identified. See for example commonly assigned U.S. Pat. No. 6,345,274, entitled “Method and Computer Program Product for Subjective Image Content Similarity-based Retrieval” filed by Zhu et al. on Jun. 29, 1998. The '274 patent describes image processing algorithms that allow images to be searched by identifying images that are like the exemplar. However, photographs typically contain many objects, shapes, textures, colors, and other visual elements that may or may not be important in the search for similar images. Therefore, algorithms that search for images based upon an exemplar, are also required to make assumptions about which elements of the image are important in order to reduce the possibility that images will be identified by the algorithms as being similar to the exemplar based upon the presence of visual elements that are not important to the searcher.
The effectiveness of such image enhancement, searching, and other image using algorithms can be enhanced where there is a positive indication as to what portions of an image contain the most important image elements. Therefore there is a need for an automatic way to determine what visual elements in an image are of greater importance.
One way to obtain an indication of what is important in an image is to obtain the indication at the time the image is captured. A variety of cameras are known which attempt to discern which areas of an image of a scene are of interest to a user of the camera. For example, U.S. Pat. No. 5,765,045, entitled “Camera Capable of Detecting Eye-Gaze” filed on Jun. 7, 1995, by Takagi et al. and Japanese Publication, No. JP 2001 116985, entitled “Camera With Subject Recognizing Function and Subject Recognizing Method” filed by Mitsuru on Oct. 12, 1999, discusses the use of the eye gaze monitoring devices in the viewfinders of the cameras described therein. The cameras described in these publications are automatic focus cameras that utilize multi-spot range finding techniques that divide a photographic scene into a plurality of spots or regions and determine a distance from the camera to each spot. The output of the eye gaze monitoring devices described therein is used to help the camera determine which of these spots are most likely to contain the subject of the image, and to focus the camera to capture images at a distance that is associated with the spot. The camera is focused at the distance from the camera to the spot identified as being most likely to contain the subject.
The use of eye gaze monitoring has also been discussed in the context of image compression in digital imaging systems. For example, U.S. Pat. No. 6,252,989, entitled “Foveated Image Coding System and Method for Image Bandwidth Reduction” filed by Geissler on Dec. 23, 1997, discusses a technique termed “foveated imaging” in which an observer's eye gaze position is monitored in real-time and communicated to a real-time image capture system that compresses the image to maintain high frequency information near the observer's point of eye gaze and discards high frequency information in regions that are not near the observer's point of eye gaze.
Thus, cameras are known that are adapted to monitor eye gaze and use information from eye gaze monitoring equipment to make decisions about the photographic or video imaging process. However, the information leading to those decisions is discarded after the image is captured. While it is known to record eye gaze position in the non-analogous art of physiological study, such studies have typically been performed by monitoring the eye gaze position of the observer and making recordings of the eye gaze travel of the observer on a medium such as a videotape or datafile that is separate from the image being observed. This creates difficulties in associating the data with the images and in preserving the association of the image with such data over the useful life of the image.
Further, it will be appreciated that while in many circumstances eye gaze monitoring may provide an indication of which elements in images are important to a user, in other circumstances, eye gaze information may not directly indicate which elements in images are important. For example a user can fixate on an object during composition in order to ensure that an image is composed to reduce the appearance of the object in the image. Accordingly, cameras that rely upon eye gaze direction to make decisions in the image capture process may make these decisions based upon erroneous assumptions about which elements of the image are important. Better imaging decisions can be made during post capture image processing where more information about the user, the scene and/or the elements of interest are available and where more time and more complex image processing algorithms can be applied to better interpret eye gaze information.
Commonly assigned U.S. patent application Ser. No. 10/303,978, entitled, “Digital Imaging System With Eye Monitoring” filed Nov. 25, 2002, by Miller et al. describes an image capture system having an eye monitoring system that stores eye information including eye gaze direction information during an image capture sequence and associates the eye information with an image captured during the image capture sequence. In certain embodiments, context information is also captured with the eye information. The context information is also associated with the image. The eye monitoring system described therein is useful for its intended purpose and has broad application. However, some consumers prefer no to use eye monitoring systems.
Accordingly, what is needed is a simpler camera and method for determining what is important in a captured image and associating information with the captured image that indicates which portions of the captured image are important.